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ABSTRACT 

This monograph summarizes outcomes of a forum 
addressing strategies to improve special education program evaluation 
efforts and orient pol icyiuakers , administrators, and advocates to the 
benefits of program evaluation as an integral part of program 
improvement- The forum also addressed barriers restricting evaluation 
of program effectiveness including resource allocation, the lack of 
technical assistance, and methodological issues- Prior to the 
meeting, forum participants were sent three background papers to 
examine and asked for a list of issues in special education program 
evaluation* During the meeting a total of 92 issues were proposed and 
organized under seven broad cluster titles: (1) resources, (2) 
methodology, (3) conceptualization, (4) program improvement, (5) 
general education reform, (6) compl iance monitoring, and (7) 
stakeholder needs (i.e., the purposes of evaluation). Priority 
solutions proposed included: development of student outcome measures 
that relate to those in general education, commitment to outcome 
variables as indicators of program effectiveness, and design of an 
evaluation system to meet the varying needs of stakeholders. 
Appendices include the participant list, the meeting agenda, forms 
and worksheets, and complete lists of issue clusters and proposed 
solutions. (DB) 
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A Forum to Examine Policy and Practices Issues 
Surrounding the Annual Evaluation of Program Effectiveness 



I. PURPOSE AND ORGANIZATION OF THE POLICY FORUM 

A. Background and Purpose of the Forum 

The meeting was based on the supposition that program improvement efforts require 
systematic evaluation in order to judge their effectiveness and provide guidance for future 
innovation. A recent study by Project FORUM at NASDSE found that compliance monitoring 
has been used almost exclusively to establish the adequacy of special education programs. While 
adherence to laws and regulation is necessary, a determination of compliance alone does not help 
professionals and advocates judge the quality of current programs or services; nor do reports and 
findings of compliance monitoring activities suggest the steps that might be taken to improve 
educational outcomes for children and youth with disabilities. These essential activities are at the 
heart of program improvement and can only be accomplished with information gleaned from 
program evaluation. 

This Policy Forum was convened to suggest strategies that can be implemented within the 
next three years to improve program evaluation efforts and orient policymakers, administrators, 
and advocates to the benefits of program evaluation as an integral part of program improvement. 
Solutions to other barriers restricting the evaluation of program effectiveness, including resource 
allocation, the lack of technical assistance, and methodological issues also were considered. 

B. Preparation for the Meeting 

Project FORUM staff worked closely with OSEP personnel to identify 15 participants for 
the Forum. Efforts were made to include researchers, consultants, state and local special 
education administrators, and other individuals with expertise or interest in program evaluation. 
The list of participants appears in Appendix A. 

To provide a literature basis for the discussion of issues and solutions, Project FORUM 
staff sent three background papers to each participant: 

o Berkowitz, S. (1993). A guide to conducting stakeholder-based evaluations for 
state agency/federal evaluation studies projects (Phase V, Deliverable 23). 
Rockville, MD: Westat, Inc. 
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o DeStefano, L. & Wagner, M. (1990). Outcome assessment in special education: 
Lessons learned (Phase II, Deliverable 23, pp. 1 -16). Rockville, MD: Westat, Inc. 

c Gonzalez, P. A. (1992). State procedures for the evaluation of special education 
program effectiveness. Alexandria, VA: National Association of State Directors of 
Special Education, Inc., Project FORUM. 

Also prior to the meeting, each participant was asked to submit a list of issues, concerns, or 
problems he or she perceived as barriers to improving or expanding the use of program evaluation 
in special education. A form was prepared for this purpose (see Appendix B). Twelve forms 
were returned containing a total of 130 issues with some duplication across participants. 

After receiving the participants' issue lists, Project FORUM compiled a master list of all the 
issues. A content analysis was performed for the purpose of clustering the issues under broad 
themes that could be addressed during the discussion of solutions. In general, the wording of the 
issues was not changed except when: a) more comprehensive wording was adopted for issues 
submitted by more than one participant, and b) the original submission contained lengthy 
examples or further explanatory material. The result of the content analysis was 86 unduplicated 
issues grouped under seven cluster titles. This list was distributed to the participants when they 
checked into the hotel with instructions to try and review the entries prior to the meeting. 

C. Process of the Meeting 

The Forum met on August 2 nd and 3 rd , 1993 at the Old Town Holiday Inn, in Alexandria, 
VA. The agenda (Appendix C) began with welcoming remarks and an orientation to the tasks 
of the Forum presented by NASDSE and OSEP staff. Then, participants introduced themselves 
to the group and cited one or two issues surrounding the evaluation of special education program 
effectiveness they found most salient. 

The rest of the morning was devoted to additions, corrections, and clarifications to the 
issues submitted prior to the meeting and to the clusters and cluster titles. Three themes emerged 
from this discussion: (a) the need to use program evaluation information to improve programs 
and services; (b) the need to examine student outcomes as an indicator of program effectiveness; 
and (c) the need to coordinate evaluation efforts with general education reform. By the end of 
this session a revised list of issues and clusters was proposed. During the lunch break, Project 
FORUM used a notebook computer to addend the issue clusters and titles and to prepare a 
corrected draft for each participant. 

The afternoon session was devoted to proposing solutions to broad questions that were 
used as cluster titles. Each participant was assigned to one of three small groups representing a 
range of evaluation expertise and administrative experience. Each group was given a list of the 
cluster titles arranged in a different order. Using worksheets provided for this purpose (Appendix 
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D), the groups were asked to begin at the top of their cluster assignment list to discuss and 
propose solutions and responsible parties. As the worksheets were completed, they were entered 
into a computer file by Project FORUM Staff. 

At the start of the second day participants were given a comprehensive list of the proposed 
solutions generated the previous day by all three groups. The participants were then encouraged 
to discuss and refine any or all proposed solutions. At the end of the meeting each participant 
was given three dots to be used to nominate those solutions with the greatest potential for 
improving the capacity of SEAs to evaluate program effectiveness. At noon, OSEP staff closed 
the meeting with a brief description of future steps to be taken with the information produced at 
the meeting. 

II. OUTCOMES OF THE MEETING 

Using a notebook computer, Project FORUM staff maintained a process account of the 
large group discussions occurring on the first and second day. The notebook also was used to 
prepare and revise working summaries and other documents for the participants to use throughout 
the meeting. 

During the morning of the first day, the participants engaged in a discussion of the issues 
submitted prior to the meeting, including the appropriateness of the clusters and cluster titles. 
As a result of this discussion seven issues were added and one was deleted for a total of 92 
issues. The cluster assignments remained the same; however, there were edits to the wording of 
five cluster titles. The final version of the cluster titles appears below along with representative 
issues from each cluster. The complete, revised list of issues appears as Appendix E. 

o Cluster A: How can we increase access to and more effective use of resources 
(i.e., money, staff, time, professional development opportunities, technical 
assistance) needed for program evaluation at the state and local level? 

In order for data collected for the purpose of program improvement to be 
effectively utilized, training and technical assistance should be available to 
stakeholders in the areas of data analysis, strategic planning, program 
implementation and evaluation 

There is a lack of fiscal resources for evaluation and data management at 
the SEA and LEA level 

o Cluster B: How can we make evaluation an integral part of educational 
improvement? 
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Evaluation is perceived as an "add-on" rather than an integral part of 
decisionmaking, management, and administration 

Evaluation efforts will increase paperwork and take more time away from 
children 

o Cluster C: What are the major conceptual issues associated with evaluation of 
educational services and outcomes for students with disabilities within the context 
of the total educational experience? 

There is a lack of consensus as to what "effective" means and how to 
determine it 

It is difficult to evaluate different programs (e.g., special education) within 
the context of the total education of the child 

o Cluster D: What are the major methodological problems associated with 
evaluation of educational services and outcomes for students with disabilities 
within the context of the total educational experience? 

There is a need to systematically collect consumer satisfaction data from 
the perspectives of students and families 

Interdistrict differences in the categorization of disabilities, placement 
criteria, and other variables confound the assessment of effectiveness across 
districts (the same applies across states) 

o Cluster E: How can we address issues in special education program evaluation 
that arise from general education reform? 

There is a need to link state and local level special education evaluation 
systems with emerging general education evaluation efforts (e.g. New 
American Standards) and the inclusion movement 

Consensus is needed on parameters that define acceptable modifications of 
"traditional" assessment processes and/or valid alternative assessment 
techniques 

o Cluster F: How can we balance and coordinate program evaluation and 
compliance monitoring? 
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There are regulations at all levels that call for paperwork, not quality 
indicators 



States need clarification on the differences between a report of compliance 
results and one of evaluation results 



G 



Cluster G: How do we address differing needs for evaluation information among 
stakeholders (e.g., accountability, program improvement)? 



There exists a misalignment between Federal and local needs for evaluation 
information 



Attempts are frequently made to have program evaluation systems serve 
both accountability and program improvement functions and these two 
functions create conflicting incentives for providing accurate and complete 
information 



During the afternoon of the first day, the cluster titles and individual issues were used 
within three small groups to guide the development of solutions. A total of 53 solutions were 
generated at that time. The largest number of solutions (17) focused on the cluster addressing 
general education reform (Cluster E, above). The large group discussion during Day 2 permitted 
refinement and clarification of these solutions. As a result, three solutions were added and four 
were deleted from the list for a total of 52. Appendix F provides a compendium of the revised 
solutions grouped under the appropriate cluster title and by small group. 

The final activity for the participants was to nominate three solutions with the greatest 
potential for improving the capacity of SEAs to evaluate program effectiveness. The dots 
appearing in the "Group" column of Appendix F represent the number of nominations a particular 
solution received from the participants. A summary of the solutions (and the responsible parties) 
with the most votes is provided below. The remaining 46 solutions in Appendix F received three 
or fewer votes. 



9 votes Develop a series of papers about what special education policy and practice can None 



contribute to the school improvement movement from our research on topics such specified 
as parent involvement, individualized education, instructional strategies, transition, 
and cross-agency collaboration. These papers should be disseminated to policy 
makers and other stakeholders and should be co-authored by regular and special 
educators. 
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7 votes All means all. A general education evaluation system must include information 

* pertaining to all students (e.g., Kentucky model). For certain students the ways of 
achieving or measuring outcomes may vary, but for the vast majority of students- 
the assessments will be the same. Likewise, outcomes for students in special 
education should parallel those of their regular education peers. 

6 votes Develop a conceptual model of educational outcomes or indicators of 
effectiveness that is congruent with models used in general education. 
(The two major problems in assessment of outcomes are: when do you include the 
assessment of outcomes and when do you accommodate outcomes in the 
evaluative process?) 

5 votes In order to address needs for evaluation information among various stakeholders, 
it is first necessary to identify stakeholder groups and then, once they are known, 
to systematically assess their information needs. While it is naive to assume that 
a single evaluation will address all these needs, it may be feasible to design a 
system or sequence of local/state/federal evaluation studies that, in combination, 
may address the constellation of needs. 

5 votes Get groups (e.g., OSEP, OERI Regional Labs, RRCs) together on a regional basis 
several times within a year in a series of training activities on program 
effectiveness. Papers or "thought pieces" to go along with the technical 
assistance. Combine and organize TA resources across categorical programs (e.g.. 
Chapter 1, bilingual). 

4 votes Stakeholders must understand and become committed to the outcomes perspective. 

Provide an orientation to the importance of assessing program effectiveness rather 
than simply the process. Need to be familiar with the importance of outcomes. 
They must lopk at results with current resources, rather than trying to create new 
resources. Tie in with general education reform legislation, since the new 
legislation is improvement oriented. 



SEA, LEA, 
National 
Goals Panel 



None 
specified 



Federal or 
state agency 



OERI, OSEP, 
RRC, 03R1 
Regional 
Labs 



Coalition for 
Citizens with 
Disabilities, 
OSEP 



III. SUMMARY AND NEXT STEPS 

A total of 92 issues were proposed by 15 participants prior to or during the first day of 
the meeting. These issues were organized under seven broad cluster titles that encompassed 
issues concerning resources, methodology, conceptualization, program improvement, general 
education reform, compliance monitoring, and stakeholder needs (i.e., purposes of evaluation). 
The participants met in small groups during the afternoon of the first day to generate solutions 
under each cluster. As a result, 46 solutions were proposed. Through a nominating process, six 
solutions under five cluster titles were selected by the participants as having the greatest potential 
for improving the capacity of SEAs to evaluate program effectiveness. 

The priority solutions nominated by the group contained a number of policy implications, 
including the development of student outcome measures that relate to those in general education; 
and a commitment to outcome variables, in addition to process, as indicators of program 
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effectiveness. Other priority solutions suggested new theoretical work to include papers 
highlighting special education's contribution to school improvement and the design of an 
evaluation system or sequence that can meet the varying needs of stakeholders. Another solution 
addressed the continuing need for technical assistance and proposed combining the resources and 
expertise of OSEP, OERI, Regional Resource Centers, and OERI Labs on a regional basis. 

Discussion between Project FORUM staff and OSEP personnel concerning the meeting 
outcomes produced a few activities that OSEP could undertake in the relatively near future to 
promote better understanding of the benefits associated with evaluating program effectiveness and 
the development and use of new evaluation strategies at the state and local level. Briefly these 
are: 

OSEP needs to map out the extent to which the six highest priority solutions are 
already being addressed. Where are these activities taking place? What resources 
are being used? What information is being made available for use by others? 
One strategy for accomplishing this is to bring together several key OSEP 
contractors (e.g., the Federal Resource Center, the National Center for Education 
Outcomes) to examine the output of the policy forum in light of their workscopes 
and discuss how to direct or re-direct the resources of these projects to support the 
recommended solutions. 

OSEP could commission a paper that would identify what regulatory and policy 
initiatives it could take to better support the program effectiveness evaluation 
requirements under the IDEA. The focus would have to be on evaluation systems 
that are based on student and program outcomes and NOT replicate the existing 
systems used for monitoring compliance with statutory procedural requirements. 
This raises a pair of related questions that also should be considered. What in the 
current monitoring system could be cut back without sacrificing necessary 
protections for children? What aspects of monitoring need to be strengthened? 
How do existing statute and regulation affect the flexibility of the monitoring 
system to make these adjustments? 

• One of the lesser known sources of information about program evaluation is the 
SAFES (State Agency Federal Evaluation Studies) program. A review of these 
projects could be undertaken in order to identify the lessons learned related to the 
evaluation of program effectiveness and to inform state and federal evaluation 
personnel of alternative approaches to the broader issues associated with program 
evaluation. 
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APPENDIX A 



Participant List 



PROGRAM EVALUATION IN SPECIAL EDUCATION 



A Forum to Examine Policy and Practice Issues 
Surrounding the Annual Evaluation of Program Effectiveness 



August 2 nd and 3 rd , 1993 



Invited Guests: 



PARTICIPANT LIST 



Nicholas Argyros 
Associate 

Education Research 

Room 876 EBA 

NYS Education Department 

Albany, NY 12234 

518/474-3882 

FAX 518/473-7737 



Marsha Brauen 

Project Director 

Westat Corporation 

1650 Research Blvd. 

Rockville, MD 20850-3129 

301/738-3668 

FAX 301/294-4475 



Fred Balcom 
Supervisor 

Special Education Section 
State Department of Education 
650 West State Street 
Boise, ID 83720-3650 
208/334-3940 
FAX 208/334-2228 



Lizanne DeStefano 
Assistant Professor 
Educational Psychology 
University of Illinois 
210 Education 
1310 South Sixth Street 
Champaign. IL 61820 
217/333-8520 
FAX 217/244-7620 
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Wayne Erickson 
Manager 

Special Education Section 

Department of Education 

812 Capitol Square Building 

550 Cedar Street 

St. Paul, MN 55101-2233 

612/296-3272 

FAX 612/296-3272 



Jim Leinen 

Educational Consultant 
Western Regional Resource Center 
Center on Human Development 
University of Oregon 
Clinical Services Building 
Eugene, OR 97403 
503/346-5641 
FAX 503/346-5639 



Susan Hasazi 
Professor 
Special Education 
University of Vermont 
Waterman Building, Room 405 
Burlington, VT 05405 
802/656-2936 
FAX 802/656-1357 



Jeri Nowakowski 
Executive Director 

North Central Regional Educational Lab 
1900 Spring Road, Suite 300 
Oak Brook, IL 60521-1480 
708/571-4700 
FAX 708/571-4716 



Connie Hawkins 
Director 

Exceptional Children Assistance Center 
P.O. Box 16 
Davidson, NC 28036 
704/892-1321 

FAX (Call to hook-up) 704/892-5028 



Ken Olsen 

Technical Assistance Specialist 
Mid-South Regional Resource Center 
114 Mineral Industries Building 
University of Kentucky 
Lexington, KY 40506-0051 
606/257-7936 
FAX 606/258-1901 



Michele Krantz 
Director 

Special Education & Staff 

Development 
Frederick County Public Schools 
7630 Hayward Road 
Frederick, MD 21702 
301/694-1052 
FAX 301/698-2736 



Robert Tobias 
Director 

Office of Research, Evaluation, 

& Assessment, Room 728 
NYC Board of Education 
110 Livingston Street 
Brooklyn, NY 11201 
718/935-3762 
FAX 718/935-5490 
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James Ysseldyke 
Director 

National Center on Educational Outcomes 

University of Minnesota 

350 Elliot Hall 

75 East River Road 

Minneapolis, MN 55455 

612/626-1530 

FAX 612/624-0879 



U. S. Department of Education: 

Lou Danielson, Acting Director 
Division of Innovation & Development 
Office of Special Education Programs 
U.S. Department of Education 
202/205-8119 
FAX 202/205-8105 

Gerrie Hawkins, Senior Program Analyst 

Program Review Branch 

Office of Special Education Programs 

U.S. Department of Education 

202/205-5386 

FAX 202/205-9179 

Ruth Ryder, Chief 

Program Review Branch 

Office of Special Education Programs 

U.S. Department of Education 

202/205-5547 

FAX 202/205-9179 



Susan Sanchez, Education Program 

Specialist 
Special Studies Branch 
Office of Special Education Programs 
U.S. Department of Education 
202/205-8998 
FAX 202/205-8105 



Observers: 

Patricia Bourexis 

The Study Group, Inc. 

1 1 Lake Park Court 

Germantown, MD 20874-5400 

301/428-0258 

FAX 301/428-3411 

Carol Valdivieso 

Academy for Education Development 
1255 23 rd Street, NW 
Washington, DC 20037 
202/862-1956 
FAX 202/466-8693 



NASDSE Staff: 

Martha Fields, Executive Director 
Trina Osher, Project FORUM Director 
Patricia Gonzalez, Policy Analyst 
Eileen Ahearn, Senior Policy Analyst 



Susan Catlett, Intern 
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APPENDIX B 

Issues Submission Form for Participants 



ISSUES SURROUNDING PROGRAM EVALUATION IN SPECIAL, EDUCATION 



From: name- 



Instructions: On this form please list any issues, concerns, or problems you perceive as 
barriers to improving or expanding the use o f program evaluation in special education. We 
are interested in any issue regardless of the level of special education administration (Le., 
Federal, state, local, or building) most directly effected. The issues may be listed in any order, 
but please try to phrase them as statements rather than questions. Attach additional sheets, 
if necessary. 



Your ideas are important to the success of this policy forum. Please return this form ON OR 
BEFORE JULY 19 th to allow Project FORUM staff sufficient time to compile input from all 
participants and organize it for efficient use at the meeting. THANK YOU. 

Please return the list to: Patricia Gonzalez, NASDSE, 1800 Diagonal Road, Suite 320, 
Alexandria, VA, 22314, Phone: 703/519-3800, FAX: 703/519-3808. 
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APPENDIX C 

Meeting Agenda 



PROGRAM EVALUATION IN SPECIAL EDUCATION 

A Forum to Examine Policy and Practice Issues 
Surrounding the Annual Evaluation of Program Effectiveness 

August 2 nd and 3 rd , 1993 



AGENDA 



Monday, August 2, 1993 

8:30 - 9:00 Continental Breakfast - (5 th Floor, Brent I) 

9:00 - 9:30 Welcome and Orientation 

♦ Welcoming Remarks 

Martha Fields, Executive Director, NASDSE' 
Trina Osher, Director, Project FORUM 
Lou Danielson, Acting Director, Division of Innovation and 
Development, OSEP 

♦ Setting the stage: Susan Sanchez, Education Program 
Specialist, Division of Innovation and Development, OSEP 

♦ Orientation to the Agenda and Logistics: Pat Gonzalez, 
Project FORUM 

9:30 - 10:30 Group Introductions 

♦ Participants will briefly describe their experiences in education 
program evaluation and identify one of their key issues 
surrounding the evaluation of special education programs 

10:30 - 10:45 ♦ ♦ ♦ Break ♦ ♦ ♦ 

10:45 - 12:15 Review and Verification of the Issues. Clusters, and Clu ster Titles 

♦ Description of the issue clustering process 

♦ Additions, clarifications, corrections to the individual issues 
submitted prior to the meeting 
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♦ Clarifications or corrections to the clusters and/or cluster titles 
12:15- 1:15 Lunch - (5 th Floor, Brent II) 

1:15 - 1:30 Orientation to Afternoon Activities 

♦ Staff recap of morning session 

♦ Small group formation and assignment 
1:30-4:15 Sm all Group Activity 

♦ The groups will be asked to suggest relatively detailed 
solutions to the issues along with the parties responsible for their 
implementation 

4:15-4:30 Reconvene Full Group 

♦ Collection of information from small groups 

♦ Description of planned activities for Tuesday 



♦ ♦ ♦ 



Tuesday, August 3, 1993 

8:30 - 9:00 Continental Breakfast - (5 th Floor, Brent I) 

9:00 - 10:30 A Discussion of the Issues and Solutions 

10:30 - 10:45 ♦ ♦ ♦ Break ♦ ♦ ♦ 

10:45 - 11:30 Selecting Promising Solutions 



♦ Solutions with the most potential for improving the capacity 
of SEAs to evaluate special education program effectiveness are 
identified for each issue 



11:30 -12:00 Conclusion and Description of Next Steps 

♦ OSEP and Project FORUM staff 
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APPENDIX D 

Worksheet for Small Group Activity 
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REVISED ISSUE CLUSTERS 

Based on Group Discussion - August 2, 1993 

A. How can we increase access to and more effective use of resources (i.e.« money, staff, time, 
professional development opportunities, technical assistance) needed for pro gram evaluation at the 
state and local level? 

Even the few evaluation resources available in general education are seldom available in special 
education because the special education is viewed as a separate system with its own set of resources, e.g. 
evaluation staff/KQ 

Lack of fiscal resources for evaluation, data management at SEA and LEA level/GH,WE,NA,MB,MK 

Not enough staff available at the state level to design and conduct evaluations and provide technical 
assistance to LEAs/GH,WE 

Not enough funding, materials, and equipment available for adequate data management systems/GH,JL 

Program evaluators are unaware of the special issues in studying special education programs/KO 

There are too many interrelated variables to permit SEAs to do a complete program evaluation with 
existing resources/FB,JL 

Time must be made available to complete evaluations/MB,MK 

Reallocation of resources to collaboration between general and special education/ADDED BY 
DISCUSSION 

Several of the issues in this cluster pertain to training needs: 

In order for data collected for the purpose of program improvement to be effectively utilized, training 
and technical assistance should be available to stakeholders in the areas of data analysis, strategic 
planning, program implementation and evaluation/SH,MB,KO 

Lack of training at local level on how to write IEPs that can be used in evaluation process/CH 

Lack of training opportunities in evaluation for teachers and administrators (state and 
local)/WE,LD,MK,KO 

Shortage of trained practitioners and service providers in evaluation/NA,MB,JL 

Special educators lack a collegial network for sharing program evaluation experiences and results/KO 

Varying levels of staff expertise greatly affects program evaluation/MK 
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B. How can we make evaluation an integral part of educational improvement? 

Evaluation is perceived as something an outside expert conducts/KO 

Avoidance of evaluation information permits the status quo to continue (no need for change)/ WE 

Building support among teachers, principals, and others for evaluation may take considerable time and 
effort/MB 

Controversy over "outcomes" (OBE) has slowed push for evaluation/CH 

Evaluation efforts will increase paperwork and take more time away from children/WE,NA,MK 
Evaluation is not an activity valued by educators/WE,CH 

Evaluation is perceived as an "add-on" rather than being integral to decision making, management and 
administration/KO,JL 

Evaluation results are frequently used for punitive purposes and therefore evaluation is perceived as 
negative/KO 

Intra and inter agency, continuous evaluations are difficult to do/ADDED BY DISCUSSION 

Lack of commitment to address program improvement through evaluation over ti'me/GH,JL 

Lack of rewards/incentives for using program evaluation information/LD 

No real desire at the state/federal level to evaluate effectiveness/JY,WE 

People do not want to have their work scrutinized or to be held accountable/WE,NA,KO,JL 

Significant stakeholders (e.g., parents, advocates, regulators, funders, taxpayers) are more concerned with 
access to services, appropriateness of services, and fiscal accountability than outcomes for students/NA 

Special educators are under the impression that program evaluation takes unique skills that they cannot 
easily learn/KO 

We are disseminating old, program and practice models that have not been evaluated for 
effectiveness/NA 
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C. What are the major conceptual issues associated with evaluation of educational services and 
outcomes for students with disabilities within the context of the total educational exp erience? 

A lack of consensus as to what "effective" means and how to determine it/JY,NA,JL 

Careful consideration must be given to how the evaluation data will be used/JL 

Clearly, evaluation drives programs so we need to be extraordinarily cautious that we have the child's 
individual needs always at the top of our priority list/MK 

Difficulty in deciding to evaluate special education generally or by category/JY 

Generalizing results from unique programs designed for individuals creates a conundrum when the intent 
is to describe a generic "special education program'VKO 

Need to define student success/GH 

How to evaluate different programs (e.g., special education) within the context of the total education of 
the child./ADDED BY DISCUSSION 

No consensus exists regarding a definition for program evaluation nor are clear distinctions made 
between program evaluation, compliance monitoring, and research/KO 

Special education is being re-defined as a service, not a program. This may necessitate different 
evaluative techniques/FB 

The terms effectiveness, quality, impact, and outcomes remain undefined/KO 

There is a lack of shared understanding regarding the Joint Committee Standards on the Evaluation of 
Educational Programs, Projects and Materials/KO 

There is a lack of consensus on the intentions of education and the expected outcomes that would accrue 
therefrom/KO 

When evaluation is primarily data or program outcome driven, the statistics can become the focus of the 
evaluation rather than the student and their outcomes - putting numbers ahead of the students/MK,JL 

How do we include students "at risk" of placement in special education?/ADDED BY DISCUSSION 
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D. What are the major methodological problems associated with evaluation of educational services 
and outcomes for students with disabilities within the context of the total educational experience? 

An assessment needs to be made of the validity and reliability of evaluation results to ascertain if they 
should be used to make program changes/MB 

Because lEPs are generic, nonspecific, and modest in scope, they do not adequately define student 
outcomes/NA 

Evaluation models need to be generated that are flexible and responsive to small and large states, urban 
and rural settings, and are cost effective/JL, CH 

Evaluations in special education have focused on accuracy over utility and thus are too frequently 
perceived as academic exercises/KO 

Formal evaluation designs (e.g., quasi-experimental designs) often are not practical for evaluations 
conducted at the local or school level/MB 

Goals and parameters are not set consistently for programs, so evaluation consistency is difficult/MK 

How can we evaluate the special education component of a student's education?/ADDED BY 
DISCUSSION 

How do we evaluate the effectiveness of education for students with disabilities?/ADDED BY 
DISCUSSION 

Instrumentation available for measuring outcomes tend to focus on academic outcomes rather than life 
role skill outcomes that are the intention of many special education programs/KO 

Program implementation should be assessed before the impact of the program since programs often are 
not implemented as expected/MB 

There are no consistent links to systems for disseminating evaluation results, e.g., using the CSPD to 
share successful practices/KO 

There needs to be valid sampling techniques available for use statewide/JL 

We need to design efficient stakeholder evaluation models that include consumers for use statewide/JL 

We need an organized way to look at evaluation that maintains the integrity of individual programs 
within systems and allows for empowerment and school-based decisionmaking/MK 
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Several of the issues in this cluster pertain to data collection: 

Data that exists on students with disabilities in the large scale databases are insufficiently flagged/KO 

Interdistrict differences in the categorization of disabilities and placement criteria (and other variables) 
confound the assessment of effectiveness across districts. The same applies across states/RT,NA,MK,JL 

National samples of "all" students systematically exclude students in residential facilities and students 
whose total programs are provided through special education services/KO 

No consistency across states or among agencies within states as to the type of data collected/JY,JL 

Population sizes of students with disabilities are so small that generalization is difficult/KO 

There is a need to know how to aggregate evaluation data without taking flexibility away from states by 
mandating a certain set of data/JL 

There is a need to identify and collect measures to assess the impact of state-level policies (e.g. changes 
in funding formulas and service delivery models) on the provision of special education services and on 
the satisfaction and outcomes of students with disabilities/SH 

There is a need to refocus data collection efforts on factors that can be manipulated to effect program 
(and individual) outcomes/SH 

There is a need to collect post-school outcome measures on students with disabilities who exit school/SH 

There is a need to systematically collect consumer satisfaction data from the perspectives of students and 
families/SH,JL 

We need to develop mechanisms for measuring outcomes that are important to all stakeholders, including 
the student/MK 
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E. How can we address issues in special education program evaluation that arise from general 
education reform? 



Consensus needed on parameters that define acceptable modifications of "traditional" assessment 
processes and/or valid alternative assessment techniques/GH,KO 

Difficulty in determining what outcomes (e.g., jobs) to hold special education accountable for/JY 

General education and special education outcomes are frequently undefined and more frequently 
unrelated, thus creating need for separate conceptual maps and separate program evaluation systems/KO 

Low or uncertain performance (achievement) standards for students with disabilities/JY,RT 

LACK OF AWARENESS OF the degree to which modifications to statewide ass e s s m e nt s are used for 
students with dioabilities/GRRT - DELETED BY DISCUSSION 

No consistency as to whether students with disabilities are included in statewide assessment 
programs/GH,RT,KO 

No consensus as to how the students with disabilities fit into the standards setting in general 
ed./JY,RT,NA 

There is a need to link state and local level special education evaluation systems with emerging general 
education evaluation efforts (e.g. New American Standards) and the inclusion movement/SH,MB,JL 

How do we prevent discrimination toward students with disabilities when using general assessment 
tests?/ADDED BY DISCUSSION 
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F. How can we balance and coordinate program evaluation an d compliance monitoring? 

As long as LEAs remain compliant they are "safe", even if programs are of poor quality/FB,JL 

Even when evaluation expectations are defined for LEAs and SEAs, SEAs and federal government have 
inadequate standards and insufficient systems for follow-up to ensure that program evaluations are 
conducted and used for decision making/KO 

Funding formulas based on paperwork/CH 

Regulations at all levels that call for paperwork not quality indicators/CH 

SEAs are held to only compliance issues by OSEP monitoring creating a lack of incentive/FB 

States need clarification on the differences between a report of compliance results and one of evaluation 
results/GH 

States need clarification on the content of the annual evaluation section of the Part B State Plan/GH 

There remains a question as to whether districts can or should focus on effectiveness issues before 
resolving compliance issues/FB 

Use of compliance monitoring as the principal evaluation strategy /JY 



ERLC 
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G. How do we address differing needs for evaluation information among stakeholders (e.g.. 



accountability, program improvement)? 



Attempts are frequently made to have program evaluation systems serve both accountability and program 
improvement functions and these two functions create conflicting incentives for providing accurate and 
complete information/KO 

Conflicts may arise between the movement toward accountability and program evaluation/MB 

Confusion exists regarding the different purposes for program evaluation, e.g., accountability, program 
improvement, public information, policy development/KO 

Misalignment between Federal and local needs for evaluation information/LD,SH 

Misalignment between Federal expectations and local capabilities in program evaluation/LD 

The results of evaluations need to be presented in a format useful to service providers and with 
recommendations useful to them/MB 



ERLC 
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List of Proposed Solutions 
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PROGRAM EVALUATION IN SPECIAL EDUCATION 

Small Group Activity - August 2, 1993 



Solutions with the most potential for improving the 
capacity of SEAs to evaluate program effectiveness ... 

CLUSTER A. How can we increase access to and more effective use of resources (i.e., money, staff, 
time, professional development opportunities, technical assistance) needed for program evaluation 
at the state and local level? 



Group 1 


Solutions 


Responsible Party 


i 


A S pflft r>f th^ rpanthoriyation of IDEA, it could be mandated that funding 
be contingent upon the provision of the data on program effectiveness. 
Program effectiveness should be clearly distinguished from program 
compliance. 

If so, states will most likely oppose because of lack of resources, so a 
proportion of the funding will be allocated toward program effectiveness 
evaluation. The money could come from compliance monitoring. 


Congress 


i 

«««« 


Stakeholders must become committed to the outcomes perspective. Provide 
an orientation to the importance of assessing program effectiveness rather 
than simply the process. Need to be familiar with the importance of 
outcomes. They must look at results with current resources, rather than 
trying to create new resources. 

Tie in with general education reform legislation, since the new legislation is 
improvement oriented. 


CCD and OSEP 


Lrg. Group 
Discussion 


Get groups (e.g., OSEP, OERI Regional Labs, RRCs) together on a regional 
basis several times over a year in a series of training activities on program 
effectiveness. Papers or "thought pieces" to go along with the technical 
assistance. Combine and organize TA resources in categorical programs 
(e.g.. Chapter 1, bilingual). 


OERI, OSEP, RRC, 
OERI Regional Labs 



1 The dots appearing in the "Group" column represent the number of "preferred solution" nominations 
from the participants. 
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PROGRAM EVALUATION IN SPFXIAL EDUCATION 

Small Group Activity - August 2, 1993 



Solutions with the most potential for improving the 
capacity of SEAs to evaluate program effectiveness . . . 

CLUSTER B. How can we make evaluation an integral part of educational improvement? 



Group 2 


Solutions 


Responsible Party 


I 


Promulgate promising practices. OSEP could generate funds for this. 

Look at other states who have implemented program evaluation and 
highlight successful practices through a conference, journal article, etc. 
(AEA) 


OSEP 


1 


Withhold a portion of district funding for a competition. Districts compete 
for funding for a program evaluation plan. This creates an incentive to come 
forward and compete for funding. So, the LEA who wins will implement 
the program. Other districts that competed will have a model to implement 
if they choose. New Jersey is a an example (extra money for districts that 
evaluate effectiveness). 


SEAs 









2 The dots appearing in the "Group" column represent the number of "preferred solution" nominations 
from the participants. 
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PROGRAM EVALUATION IN SPECIAL EDUCATION 

Small Group Activity - August 2, 1993 



Solutions with the most potential for improving the 
capacity of SEAs to evaluate program effectiveness . . . 

CLUSTER C. What are the major conceptual issues associated with evaluation of educational 
services and outcomes for students with disabilities within the context of the total educational 
experience? 



Group' j 


Solutions | 


Responsible Party 


1 

• 


Get a group of regular, special educators, and other stakeholders to identify 
the components of program effectiveness. What do we even consider to be 
effective program evaluation: child count; transition; drop out rate; student 
graduation? 


everyone (general and 
special education) 


3 

*•* 


Gain consensus around "effectiveness" and how to measure it. What needs 
to be done next is too "sell" this from a values perspective to general 
education and assure our colleagues that adaptations in curriculum, 
instruction, and the physical environment can be made in general education 
settings. 


NCEO 


3 


Develop a series of papers about what special education policy and practice 
can contribute to the school improvement movement from our research on 
topics such as parent involvement, individualized education, instructional 
strategies, transition, and cross-agency collaboration. These papers should 
be disseminated to policy makers and other stakeholders and should be co- 
authored by regular and special educators. See grid (next page) for 
examples. 











1 The dots appearing in the "Group 0 column represent the number of "preferred solution" nominations 
from the participants. 
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GRID TO EXPLAIN SERIES OF PAPERS 
Cluster C - Group 3 



Sp.Ed. Areas of 
Historical Experience 


Current Status 


Results in Spec. Ed. 
(Data Base) 


Implications for 
Regular Ed (Reg. Ed. 

Uala DaosZ) 


Ed Reform Goals 


Parental invnlvprnpnt 
r aiciiidi liivvji Ydiiciii 








Cross-agency 
collaboration on 

UdlCXll Ul ^lillUlCil 








Study & planning for 
transition ;>f students 








Individual student 
program tailoring and 
assessment 








Performance 
assessment 
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PROGRAM EVALUATION IN SPECIAL EDUCATION 

Small Group Activity - August 2, 1993 



Solutions with the most potential for improving the 
capacity of SEAs to evaluate program effectiveness ... 

CLUSTER D. What are the major methodological problems associated with evaluation of 
educational services and outcomes for students with disabilities within the context of the total 
educational experience? 



Group 4 


Solutions 


Responsible Party 


1 


States competition for monies can occur at the systems level, if LEAs knew 
they would get funding, it could help to maintain the integrity of individual 
buildings. They would feel more comfortable in having a decision cr choice 
as to what is done. 


SEAs, LEAs 


1 


Develop a conceptual model of educational outcomes or indicators of 
effectiveness that is congruent with models used in general education. (The 
two major problems in assessment of outcomes are: when do you include 
the assessment of outcomes and when do you accommodate outcomes in the 
evaluative process?) 




2 


Data should not be aggregated beyond its level of interpretability or 
meaningfuiness. Without appropriate contextual information, it is often 
difficult and possibly unethical to make meaning of aggregated data. 
"Aggregateability" should not be assumed; instead it must be carefully 
justified and empirically validated. 


OSEP 


2 
» 


Perhaps one strategy that may prove useful for communication the findings 
or impact of federal or multi-site initiatives would be to conduct meta- 
evaluations of individual evaluation reports. The meta-analyses would be 
aimed at synthesizing findings from state/local reports in ways that preserve 
important context and allow for understanding of commonalities and 
differences among local/state implementation and impact. 


OSEP 


2 
* 


In order to have true "stakeholders-based" evaluation, the design must 
permit all stakeholder groups to have equal input into the design, data 
collections, reporting, and use of the evaluation without fear of reprisal. 
Different information for making different decisions. 


Federal, state, local 
agency 


Lrg. Group 
Discussion 


Need data collection systems for policymakers at state/federal levels - a 
different kind of data for policy analysis. 





4 The dots appearing in the "Group" column represent the number of "preferred solution" nominations 
from the participants. 



I mat Report l'ohc\ I'urum on the Evaluation of Program l-flirttveness 
Protect hORVkl at SASHSr 



38 



I'axe i6 
Xfarch UK 1994 



Lrg. Group 
Discussion 


Policy forum to discuss what information is needed by policymakers, who 
are the policymakers, and how do we meet state and national education 
goals. 




3 
• 


Allow 3-5 states to have waivers for a reduction in compliance data to 
promote use of resources for outcomes/services evaluation. Waivers would 
.be based on criteria that indicate strong former compliance (access and 
procedures). Application ror waiver process wouiu require Key siaKenoiuer 
involvement and approval. 


OSEP and selected 
states and LEAs 




Consider from an organizational perspective locating evaluation and 
compliance monitoring in one unu. 


OSEP and SEAs 


3 


Results of program evaluation will not be utilized in a punitive fashion, but 
rather will be used to facilitate programmatic change through ongoing 
improvement plans, professional development, and technical assistance. 


OSEP, LEAs, and SEAs 


3 


Position paper that describes and analyzes the state of evaluation in special 
education and makes recommendations regarding evaluation design that can 
inform a variety of stakeholders including policymakers and consumers. 


Project FORUM or 
some other contractor 
of OSEP if FORUM 
isn't appropriate 


3 
• 


A joint initiative by OERI & OSEP to analyze the findings of OSEP funded 
field initiated and target research projects in the areas of family 
involvement, alternative assessment, effective instructional practices, and 
other topics relevant to the inclusion of students with disabilities into the 
general school reform movements. 


OSEP and OERI 


3 


Expand Section 618 authority to expand technical assistance to enable all 
states to apply 


Congress - requires 
change in legislation to 
allow non-funded states 
to receive TA 


3 


Consider joint research and evaluation initiatives between OERI and OSEP. 

Have OERI and OSEP develop a joint position paper on the relationship 
between special education and general education reform. 


Explore mechanisms 
such as intra-department 
agreements or 
legislative mandate 


3 


Disseminate syntheses of high quality evaluation methods and findings 
conducted by LEAs and SEAs. 


Consider joint 
dissemination between 
RRCs and OERI funded 
centers 


3 
* 


Develop a set of explicit standards and broad outcomes at both the student 
and service system levels that interface with general education standards 
associated with school reform efforts and that take into account the 
adaptations needed for individual students. 


NAEP, OERI, OSEP, 
SEAs, PTI, national 
goals panel, NCEO, 
CCSSO 


3 


Provide increased amount of money for the study of specific problems of 
national relevance for a limited scope of time. 


OSERS and N1DRR 
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PROGRAM EVALUATION IN SPECIAL EDUCATION 

Small Group Activity - August 2, 1993 



Solutions with the most potential for improving the 
capacity of SEAs to evaluate program effectiveness . . . 

CLUSTER E. How can we address issues in special education program evaluation that arise from 
general education reform? 



Group 5 


Solutions | 


Responsible Party 




i 
i 


Project FORUM needs to convene group of people representative of the 
disability community to come to agreement on how they want special 
education students included in performance standards. Look at the whole 
"standard setting business" and how we want students with disabilities 
included. 


NASDSE 


• 


2 


All means all. A general education evaluation system must include 
information pertaining to all students (e.g., Kentucky model). For certain 
students the ways of achieving or measuring outcomes may vary, but for the 
vast majority of students the assessments will be the same. Likewise, 
outcomes for students in special education should parallel those of their 
regular education peers. 


SEA, LEA, national 
goals panel (NESIC 
reports should address 
this) 


• 


2 


Outcomes and standards should be defined and measured in ways that are 
appropriate to a diverse student population. The processes used to set 
standards should enable participation of representatives of general and 
special education, LEP, racial and ethnic minorities, gifted and talented 
programs and other special populations. 


SEA, LEA 


• 


2 


Goals and objectives on a student's IEP should be clearly linked to agreed 
upon goals for learning or standards for all students. When used in this 
way, the IEP process can serve as a means for monitoring student progress 
toward achieving meaningful outcomes. 


NESIC, LEA, SEA 


• 


2 
• 


There is need for a systematic program of research examining the effects of 
assessment modification on student performance (especially for 
performance assessments) and the relationship between the assessment and 
the agreed upon outcomes. 


OSEP 


• 


3 


There is a need for more explicit and rigorous evaluation requirements that 
address program improvement as well as compliance. 







■ The dots appearing in the "Group" column represent the number of "preferred solution" nominations 
from the participants. 
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3 


Allow 3-5 states to have waivers for a reduction in compliance data to 
promote use of resources for outcomes/services evaluation. Waivers would 
be based on criteria that indicate strong former compliance (access *nd 
procedures). Application for waiver process would require key stakeholder 
involvement and approval. 


OSEP and selected 
states and LEAs 


3 


Consider from an organizational perspective locating evaluation and . 
compliance monitoring in one unit. 


OSEP and SEAs 


■> 


Resules of program evaluation will not be utilized in a punitive f ashion, but 
rather will be used to facilitate programmatic change through ongoing 
improvement plans, professional development, and technical assistance. 


OSEP, LEAs, and SEAs 


*> 


Position paper that describes and analyzes the state of evaluation in special 
education and makes recommendations regarding evaluation design that can 
inform a variety of stakeholders including policymakers and consumers. 


Project FORUM or 
some other contractor 
of OSEP if FORUM 
isn't appropriate 


-» 

:> 


A joint initiative by OERI & OSEP to analyze the findings of OSEP funded 
field initiated and target research projects in the areas of family 
involvement, alternative assessment, effective instructional practices, and 
other topics relevant to the inclusion of students with disabilities into the 
general school reform movements. 


OSEP and OERI 


■> 


Expand Section 618 authority to expand technical assistance to enable all 
states to apply 


Congress - requires 
change in legislation to 
allow non-funded states 
to receive TA 


3 


Consider joint research and evaluation initiatives between OERI and OSEP. 

Have OERI and OSEP develop a joint position paper on the relationship 
between special education and general education reform. 


explore mechanisms 
such as intra-department 
agreements or 
legislative mandate 


3 


Disseminate syntheses of high quality evaluation methods and findings 
conducted by LEAs and SEAs. 


consider joint 
dissemination between 
RRCs and OERI funded 
centers 


3 


Develop a set of explicit standards and broad outcomes at both the student 
and service system levels that interface with general education standards 
associated with school reform efforts and that take into account the 
adaptations needed for individual students. 


NAEP, OERI, OSEP, 
SEAs, PT1, national 
goals panel, NCEO 


3 


Insure that state and national standards (e.g., assessment, core curriculum) 
include the needs of students receiving special education services in all 
aspects of design and implementation. 


OSERS, OERI, NAEP, 
USDE, CCSSO, NGA 


3 


Provide increased amount of money for the study of specific problems of 
national relevance for a limited scope of time. 


OSERS and NIDRR 



NOTE: The majority of solutions for Cluster E on this page also appear in Cluster D. 
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PROGRAM EVALUATION IN SPECIAL EDUCATION 

Small Group Activity - August 2, 1993 



CLUSTER F. 
monitoring? 



Solutions with the most potential for improving the 
capacity of SEAs to evaluate program effectiveness ... 

How can we balance and coordinate program evaluation and compliance 



1 

Group" 1 


Solutions 


Kesponsiuie rany 


2 


Information collected for monitoring purposes should have a theoretical as 
well as an empirical relationship to desirable and agreed upon program 
outcomes. There is an obligation for those establishing regulations to 
demonstrate these relationships. 


Federal and state 
agency 


2 


Let's fund a study where we look at the relationship between elements of 
monitoring and student outcomes (effectiveness). The results of this study 
could be used to design a second generation study where factors that have 
been shown to be related to outcomes are incorporated into a monitoring 
system that could be piloted in some volunteer sites. Advocacy groups and 
parents should be engaged in the "re-regulation" process to allay their 
concerns. 


OSEP, CCD, national 
parent organizations 


*> 


Federal and state agencies should develop delivery or process standards for 
evaluation to clearly differentiate among compliance monitoring, 
accountability, evaluation, and evaluation for program improvement. 


Federal and state 
agency 









6 The dots appearing in the "Group 1 ' column represent the number of "preferred solution" nominations 
from the participants. 
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PROGRAM EVALUATION IN SPECIAL EDUCATION 

Small Group Activity - August 2, 1993 



Solutions with the most potential for improving the 
capacity of SEAs to evaluate program effectiveness . . . 

CLUSTER G. How do we address differing needs for evaluation information among stakeholders 
(e.g., accountability , program improvement)? 



Group 7 


Solutions 


Responsible Party 


2 


Funding requirements are driving the way evaluation is conducted by SEAs. 
The only federal dollars that don't require state match are monitoring 

rlnlldrc' c n mnnitnrino tn^pc nrprpHptirp nvpr nrncrram imnrnvpmpnt Onp 

(JUIloTb, jU 111U11 ILUI Hit! IdrvCo L/lC^vUCII^V UVC1 piUglcull lllipi \J 

solution to this problem is for OSEP to acknowledge and support both 
evaluations for monitoring purposes and evaluation for program 
improvement. The regulations should target no-match funds for both types 
of evaluation activities. Technical assistance should be available for both. 


OSEP 


• 


Cnpi ic ftinrlino an/H timplin^c fnr pvalimtinn QhnnlH VJ*rv nvpr thf* llfip OT thf* 
rUCUb, lUUUlllfci dllll lllllCllllCo lUi CVdlUCHlUII ollUUlli VCUjr UVCl UIW lilt* \J 1 U1V 

project. In the first year of funding, program evaluation activities would 
emphasize needs assessment and improvement and would not be used for 
accountability. In subsequent years, as the state program matures more and 
more, accountability aspects would be employed. Technical assistance 
would vary according to the developmental level of the project. 


Federal or state aeencv 


2 


Federal and state expectations for evaluation should be stated clearly in an 
RFP, detailing purposes for the evaluation, intended uses of and rules 
governing access to the data, anticipated costs and time demands, time lines, 
and if possible, guidelines for the design of the evaluation and procedures 
for monitoring its conduct. 


Federal or state agency 


2 
♦ 


In some instances, if the goal is to foster evaluation for self reflection and 
improvement and to remove the more threatening aspects of evaluation, the 
funding agency may find it preferable to audit the process used to carry out 
the evaluation rather than focusing on the findings of the evaluation. In this 
case, the findings will be reserved for use at the program level. 


Federal or state agency 


2 


Stakeholder identification processes should insure participation of a diverse 
group of parents, peers, professionals, and policy makers. It is particularly 
crucial to enable participation of under represented groups and to refrain 
from practices that permit limited or token participation. 





7 The dots appearing in the "Group" column represent the number of "preferred solution" nominations 
from the participants. 
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In order to address needs for evaluation information among various 
stakeholders, it is first necessary to identify stakeholder groups and then, 
once they are known, to systematically assess their information needs. 
While it is naive to assume that a single evaluation will address all these 
needs, it may be feasible to design a system or sequence of 
local/state/federal evaluation studies that, in combination, may address the 
constellation of needs. 


Federal or state agency 




The process for assessing needs in designing evaluation should allow for 
repeated reconvening of stakeholder groups to develop common definitions 
complementary methodologies, and should be conducted in the context of ' 
other evaluation and reform activities within the state or locality. 


Federal or state agency 
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